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TECHNICAL  REPORT  SUMMARY 

Virtual  Reality  is  a  marriage  between  three  dimensional  computer  graphics  and  complex  dynamical  simulation  to 
produce  a  fully  immersive,  interactive  environment.  Users  can  move  and  interact  in  this  environment  as  if  they 
were  actually  a  part  of  it.  Medicine  and  surgery  in  particular  have  been  identified  as  some  of  the  most  promising 
applications  of  this  technology  .  Virtual  reality  systems  can  be  used  to  teach  surgical  anatomy,  diagnose  surgical 
problems,  plan  operations,  simulate  and  perform  surgical  procedures  (telesurgcry).  and  predict  the  outcomes  of 
surgery.  New  Leaf  Systems  has  developed  a  highly  interactive  simulation  of  open  cholecystectomy  in  a  fully 
immersive  virtual  reality. 

Technical  Problem 

The  goal  of  the  project  was  to  provide,  on  a  rapid  prototype  basis,  a  working  model  of  virtual  abdominal  surgery. 
The  gall  bladder  has  been  selected  as  an  appropriate  organ  for  modeling  both  because  of  the  high  number  of  gall 
bladder  procedures  performed,  and  because  of  suitability  of  the  relevant  anatomy  to  the  technology.  The  open  rather 
than  laparoscopic  procedure  was  modeled  for  several  reasons:  Other  projects  are  currently  underway  to  model 
laparoscopic  procedures.  The  open  procedure  lends  itself  to  generalization  to  other  procedures.  Many  surgical 
residents  do  not  see  an  open  procedure  during  their  training  until  a  problem  has  arisen  in  a  laparoscopic 
cholecystectomy. 

Methodology 

A  literature  review  was  conducted  covering  surgical  theory  and  practice  for  'he  cholecystectomy  procedure  as  well  as 
computer  science  research  efforts  in  surgical  simulation.  In  addition  ongoing  research  projects  in  surgical  simulation 
at  other  institutions  were  surveyed. 

A  procedural  analysis  of  the  open  abdomen  cholecystectomy  operation  was  performed,  in  which  the  relevant 
anatomical  structures  and  discrete  surgical  interactions  were  identified.  The  actually  VR  cholecystectomy  simulation 
was  then  designed.  Three  dimensional  models  of  organs,  instruments,  and  the  operating  room  environment  were 
constructed  in  Swivel  3D  Pro.  Interactions  were  then  designed  in  Body  Electric.  First  library  routines  were  put  in 
place  to  map  incoming  data  coming  from  the  DataGloves  and  Tracker  to  the  virtual  head  and  hands  in  the  3D  model. 
Interactions  were  designed  to  allow  the  hands  to  hold  and  actuate  virtual  tools,  and  for  the  tools  in  turn  to  act  upon 
the  virtual  organs.  Then  the  steps  of  the  operation  were  implemented  according  to  the  procedural  analysis.  Many 
iterations  were  involved  in  this  process.  Three  dimensional  models  were  redesigned  as  new  interaction  requirements 
arose.  Interactions  were  refined  to  balance  visual  accuracy  with  ease  of  use. 

Technical  Results 

Subjects  were  drawn  from  several  categories:  two  computer  scientists,  six  middle  and  high  school  students,  two 
doctors,  one  biomedical  engineer,  and  one  artist.  All  but  one  student  were  able  to  successfully  complete  the 
procedure  within  15  minutes  on  their  first  trial.  Average  time  to  completion  was  8.93  minutes.  This  was  judged  to 
reflect  highly  successful  user  interface  design. 

The  surgeons  evaluation  was  that  the  simulation  was  useful  for  medical  students  first  learning  the  surgical 
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procedure,  who  need  to  memorize  the  steps  in  the  operation.  They  felt  that  tactile  feedback  and  most  importantly, 
accurate  tissue  deformation,  was  necessary  for  teaching  surgical  residents,  who  are  expected  to  know  the  procedure 
and  be  working  on  specific  techniques. 

Non-surgeons  including  novice  computer  users  and  students  found  the  simulation  understandable.  They  were  able  to 
see  and  identify  the  anatomical  structures  involved  and  understand  the  steps  in  the  procedure.  Lack  of  stereopsis  and 
tactile  feedback  made  some  target  acquisition  tasks  more  difficult. 

Computer  Scientists  evaluating  the  system  indicated  that  this  was  the  most  complex  set  of  user  interactions  they  had 
seen  in  a  fully  immersive  virtual  reality. 

Implications  for  further  research 

Many  questions  remain  about  how  virtual  reality  technology  can  best  be  used  in  medical  education,  after  completion 
at  New  Leaf  Systems,  this  simulation  will  be  transferred  to  the  University  of  California  at  San  Diego  Medical 
School's  Learning  Resources  Center  where  it  will  be  integrated  into,  and  evaluated  as  part  of  their  Multimedia 
System  for  medical  education.  While  highly  successful  in  meeting  its  original  design  challenges,  this  simulation 
needs  several  features  in  order  to  offer  greater  utility  to  surgeons  and  students.  Chief  among  these  are  rendering  of 
volumetric  data  (such  as  images  from  MRI  scans)  and  tactile  feedback.  A  second  and  third  year  contract  with  UCSD 
is  contemplated  in  which  these  areas  would  be  addressed. 
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APPARATUS 

The  simulation  hardware  system: 

•  Silicon  Graphics  Onyx  Reality  Engine 

•  Macintosh  Quadra  Work  station 

•  Thin-net  Ethernet  between  SGI  and  Mac 

•  Liquid  Image  Head  Mounted  Display 

•  VPL  Right  and  Left  DataGloves  with  Data  Acquisition  and  Transmission  Unit 

•  MacAdios  Data  Acquisition  Board 

•  Ascension  Flock  of  Bird  Tracking  system  with  three  sensors  (deployed  to  head  and  both  hands)  with 
large  magnet 

•  Roland  220  Sampler  for  audio  feedback 

The  Simulation  Software: 

•  Body  Electric  1.5.1 

•  Swivel  3D  Pro  2.04 

•  Isaac  1.7.0 

•  Voice  Navigator 

The  Simulation  Data  Files 

The  total  simulation  consists  of  many  types  of  data.  These  data  are  stored  in 

•  GB1 1 1  -  A  Swivel  3D  Pro  file  of  the  3D  model  (see  appendix  C  for  listing) 

•  GB 1 1 1  .DM  -  A  Body  Electric  file  of  the  Dynamics  (see  Appendix  E  for  listing) 

•  gb.prop  -  Text  file  of  Texture  Map  Configuration  (See  Appendix  D  for  listing) 

•  Audio  Sample  files 

Click 

e 

Breathing 

Cut 

Proximity 

•  Texture  Map  Files  (.rgb  extensions)  individual  bit-mapped  images  assigned  to  objects  by  gb.prop 

pSideWall . rgb  ' 

sClockWall . rgb 

Floor . rgb 

pEackKal 1 . rgb 

Drape . rgb 

sFat . rgb 

HepLig . rgb 

Belly . rgb 

Stomach. rgb 

GE.rgb 

Pancreas . rgb 
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DESCRIPTION  OF  THE  SIMULATION 
Three  Dimensional  Model 

Polygonal  3D  geometric  models  of  the  anatomy,  the  surgeon's  hands,  and  the  OR  environment  were  constructed 
primarily  in  Swivel  3D  Pro  on  the  Macintosh  Quadra.  One  organ,  the  stomach,  was  imported  from  the  SwivelArt 
anatomy  collection  developed  by  View  Point.  In  addition,  some  modeling  was  done  in  MacroModel  and  imported  to 
Swivel3D. 

The  objects  in  the  simulation  are  rigid.  Deformation,  such  as  the  stretch  or  shrink  of  the  cystic  duct  under  tension 
from  the  forceps  pull,  is  accomplished  by  morphing  or  interpolating  rendering  between  two  end  state  forms:  fully 
stretched  or  fully  shrunk,  in  the  case  of  the  duct.  Morphing  behavior  is  controlled  dynamically  in  real  time  by  the 
Body  Electric  Dataflow  network.  Thus  user  action  can  trigger  and  control  the  degree  of  morph  by  their  actions  either 
directly  on  the  organ  object  or  by  manipulation  of  a  virtual  tool  which  in  turn  acts  on  the  object 

The  models  were  rendered  in  real-time  gauroud  shading  by  Isaac  on  a  Silicon  Graphics  Onyx  Reality  Engine. 

Texture  maps  are  tiff  (bit  mapped)  files,  either  created  in  Adobe  PbotoShop  (textures  on  the  organs)  or  captured  as 
video  or  PhotoCD  images  in  the  liver  transplant  operating  room  at  the  University  of  California  .  San  Francisco,  and 
edited  in  Adobe  Photoshop  on  the  Mac. 

Anatomical  structures 

•  Liver 

•  Gallbladder 

•  GaUBladder/Liver  Fossa 

•  Cystic  Duct 

•  Common  Duct. 

•  Cystic  Artery 

•  Hepatic  artery 

•  Fatty  Tissue 

•  Gastro-hepatic  ligament 

•  Stomach 

•  Pancreas 

The  surgeon's  hands,  and  surgical  tools  are  fully  interactive  elements.  The  immediate  operating  environment,  such 
as  the  lighting  and  OR  surroundings  are  simulated  to  provide  context  and  scale  without  compromising  performance. 

Components  of  the  operating  room  environment 

•  Adjustable  Operating  Theatre  Lamps  -  Both  are  interactive .  grabbable  solid  model  objccts  and  local 
light  sin  the  computer  graphic  lighting  model.  The  simulation  user  can  grab  the  lamp  handle  and 
readjust  die  lighting  on  the  patient 
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•  Operating  Table 

•  A  simple  body  form  is  texture  mapped  with  a  photo-realistic  belly  skin  image 

•  Body  is  draped  with  blue  cloth,  texture  mapped  with  drapery 

•  Walls  of  the  operating  room  are  texture  mapped  with  photo-realistic  images  from  a  real  operating 
room 


Figure  1 :  The  virtual  operating  room  showing  texture  mapped  walls,  floor, 
drapery  .and  belly 


Dynamics 

The  interaction  dynamics,  or  behavior,  was  designed  in  Body  Electric,  a  visual  programming  language  for  virtual 
reality.  Dynamics  are  the  active  behavior  of  the  simulation  based  on  user  action.  A  listing  of  the  Dataflow  Modules 
(called  DMs)  comprising  the  network  is  included  at  the  end  of  this  report.  Page  one  "GB 1 1 1  .DM"  is  the  top  net.  in 
from  which  all  of  the  nested  DMs  both  for  debugging  and  running  the  simulation  are  called.  Subsequent  nested 
DMs  are  listed  after  the  DM  from  which  they  are  called. 

MPJThumbcalib 
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m  humbtoT  r« 
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- p/  PassRa W  T-RightOut«rj  - 

IK  Exp  Catib 

Figure  2:  a  portion  of  the  Body  Electric  Data  Massage  (DM)  network 
mapping  inputs  from  the  DataGlove  (port2-1  and  port2-2)  to  the  joints  of 
the  thumb  using  exponential  calibration 
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Interactions 

In  the  real  operating  room,  tools  are  placed  in  the  surgeon's  hand  by  a  scrub  nurse  as  they  are  requested.  In  the 
simulation,  the  user  receives  the  tool  by  making  a  special  tool  request  gesture  (cupped  hand,  palm  up)  while  giving  a 
verbal  command.  Another  gesture  exaggerated  open  band>  causes  the  tool  to  return  to  its  storage  position. 


p-RiqhtOut 

p-RiqhtOut 

P-RiqhtQut 

p-RiqhtMid 


p-RiqhtOute 


Right  Free  Object  Gra 

"gopt 

NestedDM 
Gesture  \£ight  Linked  Ob  ject  G 


Figure  3:  A  portion  of  the  Body  Electric  DM  network  defining  which  finger 
positions  will  be  recognized  as  a  “grab” 


Gestures  are  transduced  by  the  DataGloves  and  magnetic  hand  trackers  and  transformed  by  Body  Electric  into 
naturalistic  motions  of  the  virtual  hands.  Virtual  tools  are  held  in  the  hands  and  used  to  effect  organs  in  the  model. 
To  obtain  a  tool,  the  user  holds  out  the  correct  hand  in  the  reception  gesture  and  requests  the  tool  verbally  by  name. 
Once  grasped,  tools  follow  the  hand  that  holds  them  in  both  position  and  orientation.  Forefinger  flexion  is  used  to 
direct  the  angulation  of  the  virtual  tod  to  give  the  user  greater  control.  Thumb  flexion  is  used  to  open  and  closed 
hinged  tools  such  as  scissors  or  clamps.  Tools  return  automatically  to  their  "home"  position  when  released. 


Figure  4:  The  virtual  hands  hold  the  kelly  damp 
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Are  forceps  closed  end  holding 
the  gestrohepetie  ligament  ?. 
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^  Are  scissors  closed  and  cutting 
through  the  gastrohepatic  lig.  ?. 
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[  ScissorEnd 
fOHepLigOrab/  Hipest 


Figure  5:  DM  network  controlling  tool  behavior.  The  HitTest  DMs 
compare  tool  locations  to  that  of  their  targets  to  see  whether  they  are  in 
position  to  be  actuated. 


Simple  newton ian  mechanics  including  gravity  have  been  modeled.  For  example,  a  tool  can  fall  when  dropped,  or 
sail  across  the  room  when  thrown.  During  the  actual  procedure,  however,  this  capability  is  generally  superceded  by 
the  gesture  control  described  above. 


Tool  interactions  offer  some  significant  challenges  in  the  absence  of  force  feedback  or  physical  constraint  on  the 
user.  A  physical  tool  is  subject  to  forces  from  both  the  hand  that  holds  it  and  the  organ  or  surface  on  which  it  acts 
and  in  turn  applies  forces  to  both  the  hand  and  the  tissue.  On  selected  portions  of  the  model  (the  cystic  duct  and 
cystic  artery,  primarily)  the  deformation  effects  of  these  forces  on  the  model  are  simulated  by  morphing  the  structure 
in  question.  The  users  own  physical  hands  however  remain  unconstrained,  and  their  position  continue  to  be 
controlled  by  the  output  of  the  magnetic  trackers.  Therefore  an  elastic  relationship  between  a  tool  and  the  hand  is 
maintained  when  the  tool  has  grasped  a  structure.  If  the  user's  hand  moves  farther  away  than  is  physically  possible, 
the  active  part  of  the  tool  remains  in  position  while  its  orientation  follows  the  user's  hand.  Audio  feedback  is  used 
to  provide  further  information  in  the  absence  of  stereopsis  or  tactile  or  force  feedback. 
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Figure  6:  This  DM  network  triggers  audio  feedback  (the  MIDI  Sustain 
Note  DM)  when  the  Gastrohepatic  ligament  is  cut  with  the  scissors 


Surgical  tools  in  the  simulation 

•  Scalpel:  cuts  the  skin 

•  Abdominal  wall  retractor:  currently  unused 

•  Harrington  (“Sweetheart  ")  retractor:  for  retracting  the  liver 

•  Scissors:  divides  the  hepatic  ligament,  cuts  the  cystic  duct  and  artery 

•  Forceps:  grasps  the  hepatic  ligament,  cystic  duct  and  artery 

•  Right  Angle  Clamp:  currently  unused 

•  ClipApplier.  applies  clips  to  cystic  duct  and  artery 

•  Kelly  Clamp:  grasps  the  gallbladder  for  dissection  from  the  liver  bed.  and  for  removal  from  the 
abdomen 

•  Bovie  Cautery:  cauterizes  the  tissues  connecting  the  gallbladder  to  the  fossa  or  liver  bed. 

Speech  Recognition 

Speech  recognition  for  tool  request  was  implemented  in  Voice  Navigator  software.  This  proved  to  require  significant 
time  to  train  the  system  to  recognize  the  requisite  vocabulary  for  each  participant  It  also  was  not  acceptably 
reliable.  Therefore,  for  user  trials,  keyboard  input  from  a  human  operator  was  used  to  replace  machine  recognition. 

Audio  Feedback 

Four  distinct  sounds  were  digitally  sampled  and  recorded  using  a  Roland  220  Sampler,  and  are  triggered  for  playback 
dynamically  by  events  in  the  virtual  world: 

•  Ventilated  Breathing  Sound  (background  noise  for  realism  -  potentially  could  be  used  for  complications 
simulations  by  varying  frequency). 

•  Proximity  Cue  Sound  (used  to  indicate  that  the  actuating  tool  end  is  within  the  prescribed  boundaries  of 
the  target  tissue. 

•  Cutting  sound,  also  used  for  cautery 

•  Clip  Application  sound  (Gick) 
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Steps  in  the  Simulation 

The  simulation  requires  that  each  step  of  the  procedure  be  performed  in  the  correct  sequence.  A  detailed  analysis  of 
the  surgical  tasks  and  perceptual  requirements  was  made  as  the  basis  for  tuning  the  simulation  to  optimize  system 
performance  for  resolution  and  frame-rate.  Library  Research  ,  observation  of  actual  surgery  and  input  by  Wendy  K. 
Brown  MD.  surgical  resident  at  Dartmouth  Medical  Cente.r  were  applied  to  determining  the  sequence.  Subsequently, 
surgeons  evaluating  the  system  suggested  further  changes,  some  of  which  were  implemented.  The  work  product  of 
this  was  a  narrative  description  of  each  task  in  a  normal  open  Cholecystectomy  procedure.  This  document  forms  the 
script  for  interaction  dynamics  described  below. 

Sup  1 

Surgeon  (user)  extends  right  hand  with  cupped  palm  facing  up.  and  states  "scalper,  and  the  scalpel  appears  in  the 
right  hand.  The  incision  is  made  by  drawing  the  blade  across  the  abdomen  starting  at  an  X  mark.  A  proximity 
sound  cue  is  given  when  the  scalpel  is  in  the  right  zone  to  make  the  cut.  A  red  line  appears  to  signify  how  far  the 
cut  extends.  In  this  first  implementation,  the  cut  can  be  made  in  only  one  place.  Future  versions  will  offer  more 
range. 

Sup  2 

when  the  incision  is  complete,  the  abdomen  morphs  open  (a  computer  graphic  term  which  means  that  it 
automatically  undergoes  a  transformation  from  the  closed  to  the  open  state).  Later  versions  may  include  more  details 
of  the  actually  retraction  process.  An  Abdominal  Retractor  has  been  modeled,  but  is  not  currently  employed. 

Step  3 

The  liver  is  retracted  up  and  out  of  the  way  either  by  the  right  hand  or  by  the  harrington  retractor.  The  retractor  is 
then  set  in  place.  Once  positioned  die  retractor  is  stationary:  held  in  place  by  an  invisible  virtual  assistant.  At  this 
point  the  gastrohepatic  ligament  is  visible  and  covering  the  lower  portion  of  the  Gallbladder. 


Figure  7  -  The  liver  is  held  in  place  with  this  Harrington  Retractor 
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Step  4 

The  user  requests  the  scissors  in  the  right  hand  and  the  forceps  in  the  left,  grasps  the  gastrohepatic  ligament  with  the 
forceps,  and  then  cuts  it  with  the  scissor.  The  proximity  sound  cue  is  given  when  the  scissors  are  in  the  cut  zone  of 
the  ligament,  a  second  sound  indicates  actual  cutting  is  taking  place.  As  each  cut  is  made,  the  ligament  shrinks 
back  somewhat 

Step  5 

The  fatty  tissue  obscuring  the  cystic  artery,  duct,  and  common  bile  duct  is  pushed  out  of  the  way  using  the  tips  of 
the  forceps  (still  in  left  hand). 

Step  6 

The  forceps  are  then  used  to  grasp  the  cystic  duct.  This  requires  considerable  skill,  especially  in  the  absence  of 
stereopsis  or  tactility.  The  proximity  sound  cue  is  given  when  the  forceps  tip  is  in  the  grasp  zone  of  the  duct. 

When  forceps  closes  on  duct  system,  the  duct  follows  the  forceps  tip  within  certain  limits  (so  the  user  can  get  a 
sense  of  the  system  by  moving  it). 

Step  7 

The  duct  is  held  in  the  stretched  position  with  the  forceps  in  the  left  hand.  The  clip  applier  is  requested  in  the  right 
hand,  and  the  two  clips  are  applied  by  activating  the  applier  with  the  thumb.  The  proximity  sound  cue  is  given 
when  the  clip  applier  tip  is  in  the  correct  region  on  the  duct.  A  distinct  second  sound  indicates  that  a  clip  had  been 
applied.  Surgeons  testing  the  system  suggested  that  two  or  more  clips  be  applied  to  the  proximal  end  of  the  duct. 
One  is  sufficient  for  the  distal  end.  This  change  may  be  implemented  in  future  versions. 


Figure8  -  The  clip  applier  prepares  to  place  a  clip  on  the  cystic  duct  as  the 
forceps  hold  it  stretched  taut. 
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Step  8 

The  clip  appiier  is  released,  and  the  scissors  are  requested  and  used  to  cut  the  cystic  duct  between  the  clips.  This  cut 
can  be  made  only  between  the  clips.  In  future  versions,  erroneous  cuts  could  be  made,  and  consequences  explored. 

Step  9 

Steps  7  and  8  are  repeated  for  the  cystic  artery. 

Step  10 

Both  scissors  and  clip  appiier  are  released.  The  Kelly  clamp  is  requested  in  the  left  hand 
and  the  Bovie  Cautery  in  the  right  hand.  Using  the  cautery,  the  Gallbladder  is  separated  from  the  bed .  As  the 
connective  tissue  is  removed,  the  Gallbladder  becomes  free.  The  Gallbladder  can  be  pulled  about  by  the  clamp 
demonstrating  simple  tissue  deformation  simulation. 
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RESULTS 

Subjects 

Subjects  were  drawn  from  several  categories:  two  computer  scientists,  six  middle  and  high  school  students,  two 
doctors,  one  biomedical  engineer,  and  one  artist. 

Evaluation  of  Results 

Criteria  evaluated  were  ability  to  successfully  complete  the  procedure,  time  to  completion  and  narrative  evaluation. 
The  latter,  narrative  evaluation,  is  considered  to  be  the  most  meaningful  criterion,  especially  in  the  case  of  evaluation 
by  surgeons. 

Time  to  Completion: 


Average  time  Students 

10:07 

Average  time  MDs 

11:36 

Average  time  Researchers 

5:28 

Average  time  to  completion  All 

8:93 

Best  Times  were  under  2  minutes. 

The  middle  school  and  high  school  students 

Perhaps  most  importantly,  participation  in  the  simulation  offered  an  opportunity  for  the  student  to  try  on  the  role 
of  surgeon,  without  the  onus  of  real  outcome  or  grades.  The  psychological  gain  may  well  be  in  the  child's 
expanded  view  of  self  and  possibilities.  In  addition ,  the  students  learned  functional  anatomy ,  the  names  of  organs 
and  were  easily  able  to  retain  the  sequence  operative  tasks.  It  is  conjectured  that  the  high  degree  of  computer 
experience  (including  video  games)  make  immersive  simulation  particular  effective  in  training  this  group,  and  their 
somewhat  older  peers,  now  in  college. 

Medical/Professional  Evaluation 

Two  Surgeons ,  Toshi  Mori  MD  and  Sunil  Bhoyrul  MD,  and  one  biomedical  engineer,  Frank  Tendick  PhD,  from 
UCSF  used  the  system  on  May  18th,  1994.  Their  evaluation  of  the  simulation  was  that  it  would  be  very  useful  for 
medical  students  first  learning  the  surgical  procedure  who  needed  to  memorize  the  steps  in  the  operation.  They  felt 
that  the  level  of  realism  was  too  low  to  be  of  use  teaching  surgical  residents,  who  are  expected  to  know  the  procedure 
and  be  working  on  specific  techniques.  This  was  especially  true  of  certain  aspects  of  the  operation.  For  example, 
both  indicated  that  the  dissection  of  the  gallbladder  from  the  liver  bed  is  a  critical  part  of  the  operation.  The 
connective  tissue  is  simulated  very  simply.  In  fact  its  representation  as  a  set  of  polygons  linking  the  liver  and 
gallbladder  is  almost  more  symbolic  of  the  process  than  descriptive  of  it.  This  kind  of  abstraction  can  useful  in 
procedural  training  =  ie  learning  the  steps  required  to  perform  the  operation,  but  is  probably  not  of  value  in  learning 
the  actual  skill  involved  in  safely  performing  the  dissection.  It  has  been  suggested  that  this  part  of  the  procedure 
(dissection  of  the  Gallbladder  from  the  liver  bed)  is  of  such  importance,  it  ought  to  be  simulated  separately  in  a  great 
deal  more  detail. 


New  Leaf  Systems,  Inc.  •  ARPA  Contract  number  N00014-93-C-0279 


Final  Report  -  A  Virtual  Reality-Based  Simulation  Of  Abdominal  Surgery  Page  14 

June  21,  1994 

Computer  science  Evaluations 

Computer  Scientists  using  the  system  were  highly  impressed  by  the  number  and  complexity  of  individual 
interactions  simulated.  This  was  less  apparent  to  doctors  and  students,  both  of  whom  tended  to  compare  the 
simulation  to  the  physical  world  with  its  limitless  range  of  interaction  potential.  This  points  out  a  major 
difficulty  in  developing  simulations  -  the  need  to  define  and  support  an  acceptable  level  of  complexity  and  variance 
in  interaction,  without  overwhelming  the  system  and  the  developers. 

User  Interface  Issues 

•  Need  for  stereopsis:  The  current  version  of  the  simulation  is  monoscopic.  This  is  a  purely  economic  issue. 
The  simulation  contains  all  the  necessary  information  to  generate  stereo  views,  but  the  Silicon  Graphics  Onyx 
lacks  the  video  splitter  internal  card  necessary  to  generate  a  second  view  NTSC  video  signal.  The  lack  of 
stereopsis  seriously  impairs  the  user's  ability  to  perform  fine  target  acquisition  tasks  such  as  grasping  a  duct  or 
artery  with  the  forceps. 

•  Need  for  Tactile  and  force  feedback:  No  tactile  or  force  feedback  devices  were  used  in  this  simulation. 

Using  audio  and  visual  feedback  concomitantly  can  offer  some  substitute  for  tactility.  In  the  simulation,  the 
proximity  sound  cue  and  the  visual-spacial  feedback  from  perspective  and  overlap  of  near  object  over  far  (the 
scalpel  being  obscured  as  it  penetrates  flesh,  for  example)  combine  to  give  orientation.  However,  a  surgical 
simulation  environment  which  would  be  useful  for  advanced  training  would  require  realistic  force  transduction 
and  reflection.  Audio  feedback  can  give  event  information  (occurrence  of  the  sound  signifies  a  state  such  as 
proximity  or  contact).  In  trying  to  substitute  audio  information  for  force  amplitude  (loudness)  may  express 
degree  of  force  ,  however,  there  is  no  clear  mapping  far  the  vector  of  the  force. 

•  Need  for  more  accurate  deformation  and  other  behaviors  of  tissues:  Tissue  deformation  was  only  represented  on 
a  minority  of  structures  in  this  simulation.  Even  within  the  limited  confines  of  an  immersive  virtual  reality 
environment  such  as  this  one  further  work  can  be  done.  A  Simulation  concentrating  solely  on  specific  surgical 
technique  such  as  dissecting  the  gall  bladder  from  the  fossa  or  liver-bed.  has  been  proposed  and  such 
concentration  would  allow  greater  accuracy  of  detail. 

•  Head-tracked  immersion  versus  "through  the  window"  simulation 

Tracking  the  user's  head  and  hands  transforms  the  experience  from  passive  viewing  to  active  participation.  It 
may  be  the  case  that  there  are  significantly  different  learning  pathways  invoked  in  an  immersive  environment 
even  when  the  experience  is  of  relatively  low  resolution.  This  is  a  subject  for  future  research. 
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FUTURE  WORK 

In  the  next  phase  of  the  project  the  simulation  will  be  taken  over  by  Dr.  Helene  Hoffmann  and  her  colleagues  at  the 
UCSD  Medical  School  Learning  Resources  Center.  The  LRC  team  has  developed  a  comprehensive  prototype  of  a 
multimedia  system  for  medical  education.  In  year  one  of  the  new  contract,  the  open  cholecystectomy  simulation 
presented  here  will  be  integrated  into  the  UCSD  multimedia  environment  for  further  evaluation  as  a  teaching  system. 

Among  the  teaching  /learning  issues  to  be  evaluated  are  the  discrimination  between  learning  to  operate  the  Virtual 
Reality  interface  (special  gestures,  symbolic  actions)  from  learning  which  is  directly  or  indirectly  transferable  to  the 
performance  of  surgery  .  This  includes  addressing  the  question  of  whether  such  a  system  might  teach  negative 
habits. 

Initial  work  will  be  performed  using  a  “through  the  window”  approach  with  a  Stereographic  monitor  display.  This 
simulation  was  developed  in  the  Body  Electric  visual  programming  language.  In  subsequent  years  two  and  three,  it 
is  expected  that  that  the  simulation  will  be  ported  to  Talisman,  New  Leaf  s  new  virtual  reality  authoring  language. 
The  greater  capacity  of  this  tool  will  support  integration  of  external  simulations,  alternative  input  and  display 
devices  and  rendering  and  interaction  with  radiologically  derived  volumetric  data  sets.  Head  mounted  and  desk 
mounted,  head-coupled  displays  will  be  used . 
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Appendix  B  •  Images  from  the  Simulation 
Attached  are  Still  images  from  the  simulation 

•  The  Operating  Room  and  Draped  "patient" 

•  Virtual  hand  holding  Kelly  Clamp 

•  Virtual  hand  holding  Harrington  Retractor 

•  Close  up  of  the  Clip  Applier  and  Ducts  prior  to  clip  application 

•  Close  up  of  the  Clip  Applier  and  Ducts  after  clip  application 
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GBlll 

,<name>',  x,  y,  z:  yaw,  pitch,  roll. 

'World*  0,  0,  0  :  0.0,  0.0,  0.0 

'WestWall '  5433,  2219,  -6621  :  270.0,  90.0,  270.0 
* EastWall '  -8079,  2219,  -6621  :  270.0,  90.0,  270.0 
' PolhmOf fset  *  20,  801,  1220  :  0.0,  0.0,  0.0 
1 PolhmAdjust '  0,  0,  0  :  0.0,  0.0,  0.0 

•LeftHand'  -229,  -556,  -507  :  347.7,  186.0,  169.3 
' LeftPalm'  10,  179,  129  :  0.0,  0.0,  0.0 

•LeftMidPinkie'  90,  10,  -225  :  0.0,  22.9,  0.0 
' LeftOuterPinkie '  0,  0,  -79  :  0.0,  13.4,  0.0 
'LeftPinkieTip*  0,  0,  -65  :  0.0,  6.7,  0.0 


'LeftPalroGrb' 

in 

43, 

-165  :  0.0,  0.0, 

■  LeftMidRing  * 

38, 

12, 

-259  :  0.0,  16.9, 

•LeftOuterRing*  0,  0,  -90  :  0.0,  17.4,  0.0 
•LeftRingTip1  0,  0,  -74  :  0.0,  8.7,  0.0 
' Lef tMidMiddle *  -19,  9,  -264  :  0.0,  0.0,  0.0 

1 Lef tOuterMiddle '  0,  0,  -101  :  0.0,  23.7,  0.0 
' Lef tMiddleTip ’  0,  0,  -86  :  0.0,  11.9,  0.0 
' Lef tMidlndex '  -76,  13,  -237  :  0.0,  2.2,  0.0 
' Lef tOuterlndex '  0,  0,  -102  :  0.0,  4.1,  0.0 
' Lef tlndexTip '  0,  0,  -83  :  0.0,  2.1,  0.0 

' Lef t IndexEnd '  -1,  0,  -68  :  0.0,  0.0,  0.0 
•LeftThumbBase'  -50,  4,  -38  :  319.0,  359.0,  326.0 
■LeftMidThumb'  0,  0,  -120  :  0.0,  11.9,  0.0 
'LeftOuterThumb'  0,  0,  -91  :  0.0,  4.0,  0.0 
•LeftThumbEnd’  0,  0,  -71  :  0.0,  0.0,  0.0 
•LeftHandDriver'  0,  0,  0  :  0.0,  0.0,  78.4 
•LeftHandArrow'  80,  106,  0  :  0.0,  0.0,  187.8 
1 LeftCntr-WallVect ’  -220,  150,  10  :  0.0,  0.0,  0.0 
'LeftHeadWallVector '  0,  0,  -10  :  0.0,  0.0,  0.0 
•Head'  522,  -1102,  -1520  :  206.2,  171.5,  176.6 
•HeadPoint'  0,  850,  300  :  0.0,  0.0,  0.0 
' LColorSmall '  67,  178,  0  :  0.0,  0.0,  0.0 
1 LColorSmall '  -70,  180,  0  :  0.0,  0.0,  0.0 
' RColor 1  147,  176,  0  :  0.0,  0.0,  0.0 
' LColor '  -141,  180,  0  :  0.0,  359.7,  0.0 
•Face’  0,  -245,  84  :  270.0,  90.0,  270.0 
•Ear'  0,  -290,  140  :  0.0,  0.0,  0.0 
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'SoundFollow3 •  560,  -4510,  -1430  :  126.5,  0.0,  0.0 
' SoundFollow4 *  2240,  -4390,  3420  :  72.0,  0.0,  0.0 
■ SoundFollow5 '  3920,  -6390,  590  :  320.5,  0.0,  0.0 
' SoundFollow'  -760,  -740,  -260  :  281.4,  0.0,  0.0 
'Eye1  0,  -362,  428  :  0.0,  0.0,  0.0 
'HeadDriver'  0,  0,  0  :  0.0,  0.0,  0.0 
' NeckPt  *  0,  1026,  0  :  270.0,  90.0,  270.0 
■RightHand'  1080,  -268,  -473  :  202.8,  107.4,  236.4 
' RightPalm'  10,  179,  129  :  0.0,  0.0,  0.0 

• RightMidPinkie *  -79,  10,  -225  :  0.0,  12.1,  0.0 
' RightOuterPinkie *  0,  0,  -79  :  0.0,  15.4,  0.0 
' RightPinkieTip *  0,  0,  -65  :  0.0,  7.7,  0.0 
•Right Pa lmGrb'  54,  43,  -330  :  180.3,  0.0,  0.0 
' RightMidRing 1  -30,  12,  -259  :  0.0,  4.2,  0.0 
•RightOuterRing'  0,  0,  -90  :  0.0,  19.6,  0.0 
• RightRingTip’  0,  0,  -74  s  0.0,  9.8,  0.0 
* RightMidMiddle •  27,  9,  -264  :  0.0,  5.6,  0.0 

•RightOuterMiddle*  0,  0,  -101  :  0.0,  31.0,  0.0 
■ RightMiddleTip '  0,  0,  -86  :  0.0,  15.5,  0.0 
1 RightMidlndex ‘  85,  13,  -237  :  0.0,  8.5,  0.0 

•RightOuter Index1  0,  0,  -102  :  0.0,  38.3,  0.0 
’ Right IndexTip’  0,  0,  -83  :  0.0,  19.2,  0.0 
•  Right IndexEnd’  -1,  0,  -68  :  0.0,  0.0,  0.0 
• RightThumbBase ’  50,  -4,  -38  :  41.0,  359.0,  34-0 
• RightMidThumb ’  0,  0,  -120  :  0.0,  12.3,  0.0 
• RightOuterThumb 1  0,  0,  -91  :  0.0,  3.9,  0.0 
•RightThumbEnd’  0,  0,  -71  :  0.0,  0.0,  0.0 
‘ RightHandDriver ’  0,  0,  0  :  0.0,  0.0,  78.4 
’ RightHandArrow 1  80,  106,  0  :  0.0,  0.0,  187.8 
’ RightCntr-HeadWallVec 1  -220,  150,  10  :  0.0,  0.0,  0.0 
• RightHeadWallVector 1  0,  0,  -10  :  0.0,  0.0,  0.0 
• SimpleHeadHand_b07_2H ’  0,  0,  0  :  0.0,  0.0,  0.0 
•GrbHitSoundVar 1  0,  0,  0  :  0.0,  0.0,  0.0 
•GrbGestureSoundVar ’  0,  0,  0  :  0.0,  0.0,  0.0 
• FlySpeedLimit ■  270,  0,  0  :  0.0,  0.0,  0.0 
•Finger1  42,  270,  75  :  352.6,  352.1,  350.3 
’Bigger*  0,  0,  0  :  0.0,  0.0,  0.0 

•globals’  80,  80,  0  :  0.0,  0.0,  0.0 

• LEFT_TOOL ’  10,  80,  0  :  0.0,  0.0,  0.0 
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‘RIGHT_TOOL‘  10,  100,  0  :  0.0,  0.0,  0.0 
'instate'  15,  80,  0  :  0.0,  0.0,  0.0 
•  debug '  0,  0,  0  :  0.0,  0.0,  0.0 
‘status'  0,  0,  0  :  0.0,  0.0,  0.0 
'morphon'  0,  1,  18700  :  0.9,  0.0,  0.0 

• morphSkinL '  100,  1,  100  :  0.0,  0.0,  0.0 
'morphSkinR'  100,  1,  100  :  0.0,  0.0,  0.0 
'morphGHepLig'  100,  10,  100  :  0.0,  0.0,  0.0 
•morphBloodLine '  100,  10,  100  :  0.0,  0.0,  0.0 
•morphDWhole '  100,  10,  100  :  0.0,  0.0,  0.0 
'morphArtWhole '  100,  1,  100  :  0.0,  0.0,  0.0 
•Camera  14,  6,  -20  :  14.4,  259.2,  0.0 
' SloppvTool '  80,  80,  0  :  0.0,  0.0,  0.0 

'Ob -Center '  68,  -252,  -504  :  12.7,  42.0,  24.5 
' ObjTEnd'  0,  0,  0  :  0.0,  0.0,  0.0 

•ObjTGrab'  -1,  -2,  549  :  357.8,  2.6,  225.2 
' VirtToolGrab'  517,  -475,  -445  :  257.9,  21.8,  283.6 
' VirtToolEnd'  1.  32,  -550  :  278.7,  54.9,  203.3 
■Floor'  170,  2160,  -270  :  270.0,  270.0,  270.0 

' NorthWal 1 '  5326,  -6426,  59  :  180.0,  0.0,  90.0 
•Tray'  1300,  1217,  -1640  :  0.0,  0.0,  270.0 

' RAC lamp Base '  310,  770,  -140  :  270.0,  0.0,  90.0 
' C 1 ipAppl ierBase '  330,  380,  -180  :  270.0,  0.0,  90.0 
•ForcepBase'  260,  50,  -140  :  270.0,  0.0,  90.0 
'Kelly Base'  270,  -290,  -140  :  270.0,  0.0,  90.0 
' ScalpelBase '  270,  -680,  -140  :  270.0,  0.0,  90.0 
' Tray2 '  -160,  -2540,  0  :  0.0,  0.0,  0.0 

' OrganBowl '  58,  360,  0  :  0.0,  180.0,  90.0 
•ScissorBase'  150,  -266,  -140  :  270.0,  0.0,  90.0 
•BovieBase'  350,  -77,  -140  :  92.0,  0.0,  0.0 
'HarringtonBase'  320,  -110,  -140  :  270.0,  0.0,  90.0 
' AbdWallRetractorBase '  110,  -410,  -140  :  270.0,  0.0,  90.0 
'Table'  423,  -124,  -1960  :  180.0,  325.7,  180.0 
•Drape'  -473,  -65,  -182  :  180.0,  0.0,  180.0 
'Body'  -700,  235,  560  :  0.0,  0.0,  270.0 
' BodyNew '  0,  0,  0  :  0.0,  0.0,  0.0 

•GHepLigGrab'  171,  285,  -72  :  180.0,  0.0,  85.8 
' FatSlidel '  -86,  80,  90  :  32.8,  347.8,  139.2 

' FatLayerl '  -139,  2,  -32  :  289.8,  73.1,  275.3 
' FatGrabl '  -97,  -22,  76  :  97.5,  86.0,  263.2 
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■HepGastLig1  41,  -59,  8  :  149.4,  275.2,  145.8 
'WholeLig1  0,  0,  0  :  0.0,  0.0,  0.0 
'MorphLig1  0,  0,  0  :  0.0,  0.0,  0.0 
■Thorax1  305,  274,  -311  :  180.0,  0.0,  90.0 
' skinR*  0,  0,  -41  :  270.0,  0.0,  0.0 

’SkinRClosed*  0,  0,  0  :  0.0,  0.0,  0.0 
‘SkinROpen*  0,  0,  0  :  0.0,  0.0,  0.0 
* cutMark2 1  -394,  0,  342  :  268.8,  0.0,  270.0 
* skinL'  0,  0,  -41  :  270.0,  0.0,  0.0 

1 SkinLClosed1  0,  0,  0  :  0.0,  0.0,  0.0 

■cutMark1  398,  -234,  4  :  270.9,  16.8,  313.9 
1 SkinLOpen 1  0,  0,  0  :  0.0,  0.0,  0.0 
■Line1  -2,  -32,  -510  :  90.0,  0.0,  0.0 

1 LineCutPt '  111,  297,  0  :  270.0,  348.5,  0.0 
■MorphLine*  0,  0,  0  :  0.0,  0.0,  0.0 

•ShortLine1  0,  8,  8  :  269.5,  0.0,  0.0 
• WholeLine 1  0,  8,  8  :  270.0,  0.0,  0.0 
■Blood1  0,  0,  332  :  0.0,  0.0,  0.0 
■Stomach1  337,  1,  -72  :  48.0,  335.4,  209.6 
■Artery1  273,  142,  -126  :  96.2,  337.4,  119.9 

■HepaticArtery 1  -55,  1,  -23  :  138.5,  340.3,  139.6 
■ArtCut1  24,  -39,  23  :  36.9,  289.5,  105.1 

' ArtCutB'  13,  -18,  -88  :  68.1,  337.2,  22.3 
’ArtCut A1  0,  0,  0  :  0.0,  0.0,  0.0 
1 ArtWhole 1  24,  -39,  23  :  36.9,  289.5,  105.1 
■WholeFlat 1  0,  0,  0  :  0.0,  0.0,  0.0 

'ArtAnchor1  67,  -12,  -28  :  48.2,  4.9,  0.3 
' ArtPull 1  -87,  -8,  9  :  90.0,  351.0,  15.3 
■WholeBent1  0,  0,  0  :  0.0,  0.0,  0.0 

■ ArtCl ipGrab2 1  18,  -28,  -100  :  67.6,  347.8,  270.0 
1 ArtClip2 1  -1,  8,  0  :  11.8,  26.2,  10.4 
■ArtClipGrabl 1  -6,  -3,  -38  :  174.4,  8.6,  107.6 
1 ArtClipl 1  3,  3,  0  :  180.0,  4.6,  36.2 
■Duct1  259,  174,  -87  :  79.6,  332.7,  126.9 

■ CommonDuct 1  -67,  34,  -27  :  106.2,  339.5,  27.2 
1 DCut 1  0,  0,  0  :  359.4,  289.0,  58.4 

■DCutB1  13,  -18,  -88  :  68.1,  337.2,  22.3 
■DCut A1  0,  0,  0  :  0.0,  0.0,  0.0 
■Whole1  0,  0,  0  :  359.4,  289.8,  58.4 
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’ DWholeFlat *  0,  0,  0  :  0.0,  0.0,  0.0 

•DuctAnchor1  40,  -6.  -49  :  50.7,  340.6,  356.9 
' DuctPull *  -45,  -1,  9  :  90.0,  347.8.  356.3 
' DWholeBent '  0,  0,  0  :  0.0,  0.0,  0.0 

• DClipGrab2 '  40,  -26,  -81  :  88.1,  337.1,  321.8 
1 CysticClip2 ‘  -12,  -9,  6  :  177.5,  8.8,  6.6 
•DClipGrabl*  -6,  0.  -35  :  175.5,  1.6,  74.4 
‘ CysticClipl 1  -2,  3,  0  :  181.8,  358.9,  1.4 
•Pancreas*  78,  593,  -98  :  272.9,  31.7,  339.1 
' LiverRotate *  383,  184.  -120  :  5.7,  120.0,  182.6 
'GBRotate '  0,  0,  0  :  0.0,  355.0,  1.3 

•gallbladder1  -1291,  -512,  421  :  131.1,  325.9,  33.0 
• BallBladder •  8,  115,  19  :  119.6,  47.7,  28.9 
• DummyParent 1  -35.  126,  16  :  25.4,  199.0,  166.1 

'CysticClipDummy *  -33,  -24,  81  :  154.9,  11.9,  45.3 
‘ ArtClipDummy '  -24,  -47,  104  :  326.4,  344  9,  292.4 
•ArtCut Dummy '  -19,  -43,  116  :  323.8,  345.6,  33.5 
1 DCutDummy '  -24,  -14,  106  :  338.1,  337.1,  22.4 
• BallJoint '  2,  104,  -4  :  158.3,  0.0,  358.3 
■GBEndHit’  0,  -59,  0  :  0.0,  0.0,  0.0 
•Liver*  2,  -80,  48  :  101.2,  357.7,  155.7 

•LiverGrab'  -19,  122,  128  :  0.0,  349.2,  291.8 
•Fossa'  -12,  30,  -8  :  270.0,  165.0,  0.0 
• AFosHit '  0,  0,  0  :  90.0,  89.0,  270.0 
' BFosHit '  0,  0,  0  s  46.0,  0.0,  0.0 

' CFosHit 1  -4,  0,  4  :  46.0,  0.0,  0.0 
'DFosHit ’  54,  2,  3  :  43.0,  0.0,  0.0 

' EFosHit  *  -3,  0,  -5  :  49.0,  0.0,  0.0 
• FFosHit '  -2,  0,  0  :  46.0,  0.0,  0.0 
•GFosHif  0,  0,  0  :  44.0,  0.0,  0.0 
' HFosHit '  26,  0,  -4  :  12.7,  0.0,  0 
'HFos-dummypt ’  8,  8,  0:0.0,  0.0 
' HFos  *  1,  2,  -45  :  359.1,  15.0, 

' GFos '  1,  2,  -45  :  359.1,  15.0,  0. 
' FFos '  1,  2,  -45  :  359.1,  15.0,  0.0 
1 EFos ’  1,  2,  -45  :  359.1,  15.0,  0.0 
' DFos ‘  1,  2,  -45  :  359.1,  15.0,  0.0 
' CFos '  1,  2,  -45  :  359.1,  15.0,  0.0 
’ BFos '  1,  2,  -45  :  359.1,  15.0,  0.0 
'AFos'  1,  2,  -45  :  359.1,  15.0,  0.0 
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' AbdWallRetractor 1  -1480,  380,  -1537  :  0.0,  0.0,  0.0 
* AbdWallHit '  0,  -65,  207  :  0.0,  0.0,  0.0 
' AbdWallBlada '  0,  -62,  195  :  90.0,  359.7,  0.0 
* AbdWallRetractorHome '  0,  0,  0  :  0.0,  0.0,  0.0 
• HarringtonGrab *  -565,  -319,  -1116  :  169.0,  326.6,  358.6 
•Harrington'  -250,  265,  -156  :  108.8,  0.0,  180.7 
•HarLiverHit'  -24,  -144,  250  :  180.0,  0.0,  0.0 
' HarringtonHome *  -512,  -147,  -1020  :  129.6,  0.0,  0.0 
'HarringtonBlade*  0,  -62,  195  :  90.0,  359.7,  0.0 
‘ HarringtonEnd ‘  70,  -58,  0  :  90.0,  0.0,  0.0 
•HarringtonVars '  0,  80,  0  :  0.0,  0.0,  0.0 
•BovieGrab'  -1140,  383,  -1298  :  0.0,  0.0,  0.0 
■Bovie'  0,  32,  -280  :  180.0,  0.0,  0.0 

•ButtonOn’  0,  -28,  -151  :  315.0,  270.0,  314.9 
•ButtonOff'  0,  0,  0  :  0.0,  0.0,  0.0 
•ButtonHit*  0,  0,  0  :  0.0,  0.0,  0.0 
'BovieHome'  7,  -35,  -281  :  0.0,  0.0,  0.0 
•Bovieshaft'  0,  0,  0  :  0.0,  0.0,  0.0 

•BovieEnd'  -2,  0,  453  :  225.0,  90.0,  135.0 
'BovieVars*  0,  80,  0  :  0.0,  0.0,  0.0 
•ScissorGrab1  -1328,  384,  -1496  :  0.0,  0.0,  0.0 
'Scissor'  -40,  40,  -200  :  0.0,  270.0,  0.0 

• ScissorFlare •  67,  0,  0  :  180.0,  0.0,  350.0 
'ScissorEnd'  18,  228,  0  :  0.0,  0.0,  0.0 
•ScissorHome'  32,  -199,  -44  :  0.0,  0.0,  0.0 
■ ScissorVars '  0,  80,  0  :  0.0,  0.0,  0.0 
•ScalpelGrab'  792,  379,  -1218  :  0.0,  0.0,  0.0 
•Scalpel'  0,  40,  -480  :  180.0,  0.0,  0.0 

• ScalpelBlade '  -22,  0,  -4  :  270.0,  270.0,  270.0 
•ScalpelEnd'  2,  -156,  0  :  90.0,  0.0,  0.0 
• ScalpelHome '  2,  -39,  -481  :  0.0,  0.0,  0.0 
•ScalpelVars '  1,  80,  0  :  0.0,  0.0,  0.0 
' KellyGrab'  1175,  380,  -1212  :  0.0,  0.0,  0.0 
•Kelly'  0,  40,  -400  :  0.0,  270.0,  0.0 

' KellyFlare •  5,  0,  0  :  180.0,  0.0,  335.0 
'KellyEnd2 •  4,  178,  -26  :  90.0,  2.0,  0.0 
'KellyFlare2 '  8,  179,  19  :  270.0,  358.0,  0.0 
• KellyHome *  5,  -395,  -40  :  0.0,  0.0,  0.0 
' Kelly End '  0,  218,  43  :  0.0,  0.0,  0.0 
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'KellyVara *  0,  80,  0  :  0.0,  0.0,  0.0 
1  ForcepGrab *  1518,  371,  -1227  :  0.0,  0.0,  0.0 
•Fox-cap'  0,  40,  0  :  0.0,  0.0,  270.0 

•ForcepEnd'  0,  0,  -401  :  0.0,  0.0,  3.0 
• ForcepFlare '  0,  -7,  0  :  0.0,  353.0,  0.0 
• ForcaptFlare2 '  0,  7,  0  :  0.0,  7.0,  0.0 
' ForcepHome 1  31,  2,  0  :  0.0,  0.0,  0.0 
•ForcepVars'  1,  80,  0  :  0.0,  0.0,  0.0 
'ClipApGrab'  1852,  336,  -1156  :  0.0,  0.0,  0.0 

'ClipApplierEnd'  0,  32,  -550  :  270.0,  89.9,  270.0 
' ClipApplier 1  0,  32,  -128  :  270.0.  270.0,  0.0 
•ClipShaft'  16,  0,  0  :  90.0,  0.0,  270.0 
'ClipLeftFlare*  80,  16,  -2  :  180.0,  0.0,  90.0 
'ClipRightFlare'  80,  -16,  -2  :  0.0,  0.0,  280.0 
'ClipApHome *  -127,  2,  -28  :  0.0,  0.0,  270.0 
' ClipAp? 1  -80,  0,  0  :  0.0,  0.0,  90.0 
•ClipApVars '  1,  80,  0  :  0.0,  0.0,  0.0 
•RAGrab1  2240,  380,  -1177  :  0.0,  0.0,  0.0 

•RAClamp'  420,  120,  -250  :  0.0,  270.0,  0.0 
' RAFlare 1  0.  0,  0  :  180.0,  0.0,  350.0 
•RAEnd'  4,  240,  -20  :  270.0,  0.0,  0.0 


' RAEnd2 '  4,  240,  20  :  90.0 

o 

o 

o 

o 

' RAC lamphome '  -420,  -250, 

-120  : 

o 

o 

o 

o 

o 

'RAVars'  0,  80,  0  :  0.0,  0.0, 

o 

o 

'  lampgrab '  1280,  -1520,  -1789  : 

226.8, 

66.4,  249.7 

' lamp '  0,  0,  570  :  0.0,  0.0, 

90.0 

•LightKnob'  -708,  0,  -188 

:  56.1, 

0.0,  75.3 

'Objectl96 '  58,  0,  145  :  0.0,  0.0,  90.1 
•bulb'  0,  0,  0  :  0.0,  180.0,  0.0 
'  lampgrab2 •  -850,  698,  -1031  :  122.1,  293.7,  256.1 
■  lainp2 '  0,  0,  570  :  0.0,  0.0,  90.0 

’ LightKnob2 '  -808,  0,  -207  :  61.8,  0.0,  16.9 
' Object200 '  58,  0,  145  :  0.0,  0.0,  90.1 
' bulb2 ’  0,  0,  0  :  0.0,  180.0,  0.0 
•  LSloppyTool '  80,  80,  0  :  0.0,  0.0,  0.0 

•LObj Center'  -66,  -245,  -542  :  276.2,  38.2,  14.1 
■LObjTEnd’  56,  -94,  -47  :  263.9,  87.4,  0.0 
' LObjTGrab'  -1,  -1,  578  :  6.4,  359.7,  84.1 
•LVirtToolGrab'  -498,  -755,  -849  :  113.4,  25.4,  47.1 
' LVirtToolEnd '  -3,  66,  -576  :  105.4,  21.6,  68.0 
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0  _lightl  1  0.000000  0.000000  0.000000  1.  000000  1.000000  1.000000 

0.000000  0.000000  0.000000  -0.676514  -0.417236  0.606934  0.000000  90.000000 

5  spot  1  bulb  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000 

0.000000  0.000000  -1.000000  0.000000  90.000000 

5  spot 2  1  bulb2  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000 

0.000000  0.000000  -1.000000  0.000000  90.000000 

6  WestWall  1  textures /pS ideWal 1 . rgb  12400.000000  7000.000000  660.000000 

0.000000  1.000000  0.000000  -420.000000  0.000000  0.000000  -1.000000  0.000000 

6  EastWall  1  textures /sClockWal 1 . rgb  10400.000000  7800.00048 8  660.000000 
0.000000  1.000000  0.000000  -2400.000000  0.000000  0.000000  -1.000000  - 

120.000000 


6  Floor  1  textures /Floor . rgb  400.000000  400.000000  400.000000  1.000000 

0.000000  0.000000  0.000000  0.000000  1.000000  1.000000  0.000000 

6  NorthWall  1  textures /pBackWall .rgb  12800.000000  8400.000000  660.000000 
0.000000  1.000000  0.000000  0.000000  0.000000  0.000000  -1.000000  240.000000 

6  Drape  1  textures/Drape.rgb  1500.000000  1500.000000  1500.000000  1.000000 

0.000000  0.000000  0.000000  0.000000  1.000000  1.000000  0.000000 

6  FatLayerl  1  textures/sFat . rgb  300.000000  300.000000  300.000000  -0.880000 

1.000000  0.580000  0.000000  0.588000  0.000000  0.809000  0.000000 

6  HepGastLig  1  textures /HepLig . rgb  400.000000  400.000000  400.000000 
0.760000  -1.000000  1.000000  0.000000  0.620000  -1.000000  0.000000  0.360000 

6  skinR  1  textures /Belly .rgb  640.000000  800.000000  660.000000  -1.000000 

0.000000  -1.000000  -105.000000  0.000000  1.000000  0.000000  -300.000000 

6  skinL  1  textures /Belly .rgb  1019.999939  980.000061  660.000000  1.000000 

0.000000  -1.000000  36.000000  0.000000  1.000000  0.000000  1620.000000 

6  Stomach  1  textures /Stomach. rgb  50.000000  50.000000  50.000000  -0.300000  - 

0.580000  0.380000  -7.200000  0.000000  1.000000  0.500000  -30.000000 
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6  Duct  0  textures /GB . rgb  50.000000  50.000000  50.000000  1.000000  0.000000 

0.620000  -3.600000  -0.120000  0.980000  -0.060000  34.000000 

6  CommonDuct  0  textures /GB . rgb  50.000000  50.000000  50.000000  1.000000 

0.000000  0.620000  -3.600000  -0.120000  0.980000  -0.060000  34.000000 

6  DCut  0  textures /GB . rgb  50.000000  50.000000  50.000000  1.000000  0.000000 

0.620000  -3.600000  -0.120000  0.980000  -0.060000  34.000000 

6  DCutB  0  textures /GB . rgb  50.000000  50.000000  50.000000  1.000000  0.000000 

0.620000  -3.600000  -0.120000  0.980000  -0.060000  34.000000 

6  DCutA  0  textures/GB . rgb  50.000000  50.000000  50.000000  1.000000  0.000000 

0.620000  -3.600000  -0.120000  0.980000  -0.060000  34.000000 

6  Whole  0  textures/GB. rgb  50.000000  50.000000  50.000000  1.000000  0.000000 

0.620000  -3.600000  -0.120000  0.980000  -0.060000  34.000000 

6  DWholeFlat  0  textures/GB. rgb  50.000000  50.000000  50.000000  1.000000 

0.000000  0.620000  -3.600000  -0.120000  0.980000  -0.060000  34.000000 

6  DuctAnchor  0  textures/GB. rgb  50.000000  50.000000  50.000000  1.000000 

0.000000  0.620000  -3.600000  -0.120000  0.980000  -0.060000  34.000000 

6  DuctPull  0  textures/GB. rgb  50.000000  50.000000  50.000000  1.000000 

0.000000  0.620000  -3.600000  -0.120000  0.980000  -0.060000  34.000000 

6  DWholeBent  0  textures/GB . rgb  50.000000  50.000000  50.000000  1.000000 

0.000000  0.620000  -3.600000  -0.120000  0.980000  -0.060000  34.000000 

6  DClipGrab2  0  textures/GB. rgb  50.000000  50.000000  50.000000  1.000000 

0.000000  0.620000  -3.600000  -0.120000  0.980000  -0.060000  34.000000 

6  CysticClip2  0  textures/GB . rgb  50.000000  50.000000  50.000000  1.000000 

0.000000  0.620000  -3.600000  -0.120000  0.980000  -0.060000  34.000000 

6  CysticClipl  0  textures/GB . rgb  50.000000  50.000000  50.000000  1.000000 

0  000000  0.620000  -3.600000  -0.120000  0.980000  -0.060000  34.000000 
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6  Pancreas  1  textures /Pancreas .rgb  3.000000  3.000000  3.000000  0.283000 

0.262000  0.922000  1.400000  -0.021000  0.680000  0.733000  1.200000 

6  LiverRotate  0  textures/GB. rgb  50.000000  50.000000  50.000000  1.000000 

0.000000  0.620000  -3.600000  -0.120000  0.980000  -0.060000  34.000000 

6  gallbladder  1  textures/GB . rgb  150.000000  150.000000  150.000000  1.000000 

0.000000  0.620000  -3.600000  -0.120000  0.980000  -0.060000  34.000000 

6  BallBladder  1  textures/GB . rgb  50.000000  50.000000  50.000000  1.000000 

0.000000  0.620000  -3.600000  -0.120000  0.980000  -0.060000  34.000000 

6  Liver  1  textures/Liver . rgb  75.000000  75.000000  75.000000  0.951000 

0.149000  -0.271000  -2.400000  0.588000  0.000000  0.809000  0.000000 

8  1.000000  0.000000  0.000000 
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nPointList  IndexPoint 


The  point  TOOL  uses  it's  local  x  and  y  values  for: 
x  -  Tool  number  in  left  hand 
y  -  Tool  number  in  right  hand 


x»l  currently  in  right  hand, 
a  tool  in  the  right  hand? 


'RightHandTool.Lib'  Page  1,  Mon,  Jun  6,  1994 


Outl  1  if  Regular 
Out2  1  if  "Sloppy” 


'Use  Reg  or  SIoppy.NestedDM  '  Page  1,  Mon,  Jun  6,  1994 


rrm 


Is  Tool  Flared? 


1  hand 

2  tool 

3  tool  # 

4  objects 

5  Toni  fin  tankeri? 


IiVM 


Setup&Move  Loop 

m 

JDM 


Not 


atGlobais 


O^p 


NestedDM 


’RightSloppyTool.Lib'  Page  1,  Mon,  Jun  6,  1994 


Th#  motion  of  the  QaMBIaddor  to  driven  off  of  tho 
Uvor.  Tho  Uvor  rotatoo  between  0  and  750,  while  tho 
CommonDuet  (paront  of  GaHBIadder)  nooda  to  go 
from  -54  to  1055.  So  tho  value  coming  from  R-Liver 
to  mnltirdtort  hv  1  47(M  and  54  to  auhtrantari 


Outputs 


’GB  Follow  Liver.NestedDM  '  Page  1,  Mon,  Jun  6,  1994 


0 

Constant 


PassRaw  y-mo 


'Reset  for  2.NestedDM  '  Page  1,  Mon,  Jun  6,  1994 


Inputs 


290  ~~r-€  -LiverRotal 
Ripped  Down 


Ou  uts 


'Reset  for  3.NestedDM  '  Page  1,  Mon,  Jun  6,  1994 


'RightRegularTool.NestedDM  '  Page  1,  Mon,  Jun  6,  1994 


etField 


SetField 


(R-Harrinatorl 

>£itOr 

1  1 

D  r 

1200 

3000  ■ 

Constant 

eater  Than 

Used  the  ORs  below  because  the 
possible  values  range  from  right 

;tant 

'Lock  Harrington.NestedDM  '  Page  1,  Mon,  Jun  6,  1994 


Inputs 


100 

Constant 


Outputs 


>■€ 


:-morphGHe) 


MoveScissors 


NestedDM 


These  are  the  wrong  philosophy. 
Instead  of  animating  changes  to  the 
model,  we  are  just  making  changes 
to  the  tree  to  make  sure  the  model  is 

rnhust  nnnunh 


'Surrogate  User.NestedDM  '  Page  1,  Mon,  Jun  6,  1994 


x-debuc 


Inputs 


rrooate  User 

loopf 

Iga 

Nested  DM 


When  the  sdesors  close  on  the  g|o| 

Hepatic  ligament,  turn  it  invisible.  «Sr~ 
When  you  leave  the  DM,  the  setfield  °u*>u 
is  turned  off,  so  the  ligament  remains 

inviftihlft  avao  aftAr  thA  scissors  ooao 


Ligament  Being  Cuts 

|c£joJ^ 

Nested  DM 


Morph  Away  the  Lig 


Requirements: 

Forceps  grab  ligament. 

Scissors  cut  while  touching  ligament. 


x-mstate 


Synopsis: 

Surgeon  gets  scissors  in  right  and 
forceps  in  left. 

Surgeon  grabs  ligament  with  forceps 
end  cuts  with  .scissors 


\dvance  to  5 


x-mstate 


'4-Cut  G.Lig.NestedDM  '  Page  1,  Mon,  Jun  6,  1994 


rim 


'MoveForcep.  Nested  DM  '  Page  1,  Mon,  Jun  6,  1994 


Clock 


Outputs 


'Pause  for  4  seconds.NestedDM  '  Page  1,  Mon,  Jun  6,  1994 


'Surrogate  User.NestedDM  '  Page  1,  Mon,  Jun  6,  1994 


'Reset  for  5.NestedDM  '  Page  1,  Mon,  Jun  6,  1994 


I  Constant  1 


Are  scissors  dosed  and  cutting 
through  the  gastrohepatic  lig.  ?. 


Scissors  near  HepLig? 


HandColor.Lib 
frightouterrirKTE^f 
v  cyo 

Library 


[  ScissorEnd ' 
IGHepLioGrab/  HitTest 


'Ligament  Being  Cut?.  Nested  DM  '  Page  1,  Mon,  Jun  6,  1994 


'Forcep  near  HepLig?.NestedDM  '  Page  1,  Mon,  Jun  6,  1994 


Inputs 


'Scissors  near  HepLig?.NestedDM  ’  Page  1,  Mon,  Jun  6,  1994 


«m  m  ■+ 


Inputs 

1  -  Objects  the  Left  tool  can  grab  in  this  state 
•  Whether  the  Left  tool  should  be  able  to  grab  right  now 

-  Objects  the  Right  tool  can  grab  in  this  state 

-  Whether  the  Riaht  tool  should  be  able  to  arab  riaht  now 


Outputs 

1  -  If  an  object  is  being  grabbed  by  the  left  tool 

2-  What  object  is  being  grabbed  by  the  left  tool 

3-  If  an  object  is  being  grabbed  by  the  right  tool 

4-  What  obinot  is  heino  orabberi  bv  the  riaht  tool 


Makes  the  tools  that  are  in  each  hand  follow  the 
index  finger. 


ToolGrabbing.Lib'  Page  1,  Mon,  Jun  6,  1994 


'The  Left  Side.NestedDM  '  Page  1,  Mon,  Jun  6,  1994 


Testing  (S 


LEFT  TOOL 


InddxPoint 


SetField 


KellyGrab 

BovieGrab 


Switch 


rnPointList  indexPoint 


The  point  TOOL  uses  it's  local  x  and  y  values  for: 
x  -  Tool  number  in  left  hand 
y  -  Tool  number  in  right  hand 


Tool  currently  in  left  hand. 
Is  a  tool  in  the  left  hand?. 


'LeftHandTool.Lib'  Page  1,  Mon,  Jun  6,  1994 


cm  n  ir. 


NestedOM 


'LeftSloppyTool.Lib'  Page  1,  Mon,  Jun  6,  1994 


0  flesh 

1  red 

2  yellow 

3  areen.. 


flesh 


Inputs 


ellow 


This  DM  provides  feedback  by  changing  the  palm 
color  from  flesh  to  Green,  Yellow,  or  Red. 


Outputs 


'HandColor.Lib'  Page  1,  Mon,  Jun  6,  1994 


Leading  Edge 


If  the  machine  state  changes  and  the  new  state 
isn't  8,  10,  14  then  output  a  zero  to  make  the 
existing  tool/organ  be  dropped. 


'Drop  the  tool?.NestedDM  '  Page  1,  Mon,  Jun  6,  1994 


GetField 


'Reset  for  6.NestedDM  '  Page  1,  Mon,  Jun  6,  1904 


This  lets  the  tool  tip  El - 

move  objects  around.  Outputs 


'LeftRegularTooLNestedDM  '  Page  1,  Mon,  Jun  6,  1994 


mm 


'Is  Tool  Flared?.NestedDM  '  Page  1,  Mon,  Jun  6,  1994 


'Set up& Move  Loop.NestedDM  '  Page  1,  Mon,  Jun  6,  1994 


Library 


'setup.NestedDM  1  Page  1,  Mon,  Jun  6,  1994 


'Reset  for  7.NestedDM  '  Page  1,  Mon,  Jun  6,  1994 


'Reset  for  8.NestedDM  '  Page  1,  Mon,  Jun  6,  1994 


Make  clips  invisible. 


SetField 


'Reset  tor  9.NestedDM  1  Page  1,  Mon,  Jun  6,  1994 


Turn  the  whole  morph  parent  visible 
(making  the  children  but  not  the  clips  invisible) 


'Reset  for  lO.NestedDM  '  Page  1,  Mon,  Jun  6,  1994 


’Morph  Away  the  Lig.NestedDM  '  Page  1,  Mon,  Jun  6,  1994 


Synopsis: 

Surgeon  drops  scissors. 

Surgeon  pulls  away  3  fat  layers  with 
fonceos  in  seauence. 


Outputs 


Requirements: 

Forceps  touch  fat  and  move  from  X  to  Y, 
making  the  layer  invisible. 

True  for  all  three  lavers. 


turn  off  sloppy 


H  -ForcepVaH 

at  5:Can  I  advance? 


[  x-mstate  1 5  1 


Nevf  Move 


OOO 


NestedOM 


i r— (x-ForcepVari 
turn  on  sloppy] 


Forcep  near  fatlavers  _______ 

looor  [LeftOuterRi 


land  Colors 


NestedDM 


NestedDM 


'5-Move  Fat.NestedDM  '  Page  1,  Mon,  Jun  6,  1994 


Outputs 


Inputs 

Move  forceps  from  one  hittest  to  the  next. 


90 

Constant 


{z-FatSlidel  ] 


The  above  constants  are  a  cheat  We  should 
actually  move  the  tools  for  debug  purposes, 
not  just  jump  to  the  end  result. 


'Surrogate  User.NestedDM  '  Page  1,  Mon,  Jun  6,  1994 


'Forcep  near  fatlayere.NestedDM  '  Page  1,  Mon,  Jun  6,  1994 


OZZZZD 


'New  Move  FalNestedDM  '  Page  1,  Mon,  Jun  6,  1994 


Inputs 

1  •  Objects  the  Left  tool  can  grab  in  this  state 

2  •  Whether  the  Left  tool  should  be  able  to  grab  right  now 

3  •  Objects  the  Right  tool  can  grab  in  this  stale 

4  -  Whether  the  Rioht  tool  should  be  able  to  arab  rioht  now 


The  Left  Side 


Outputs 

1-  If  an  object  is  being  grabbed  by  the  left  tool 

2-  What  object  is  being  grabbed  by  the  left  tool 

3-  If  an  object  is  being  grabbed  by  the  right  tool 

4-  What  nbiact  is  baton  orahhari  hv  tha  rioht  tool 


Makes  the  tools  that  are  in  each  hand  follow  the 
index  finger. 


ToolG^abbing.Lib,  Page  1,  Mon,  Jun  6,  1904 


Inputs 


Outputs 


'Reset  for  II.NestedDM  1  Page  1,  Mon,  Jun  6,  1994 


'Reset  for  12.NestedDM  '  Page  1,  Mon,  Jun  6,  1994 


(  KellvFlare 

3350  Y~ 
Constant 


SetField 


Outputs 


'Reset  for  13.NestedDM  '  Page  1,  Mon,  Jun  6,  1994 


Make  connective  tissue  appear. 


SetFieid 


SetField 


SetFieid 


SetField 


0 

Constant 


SetFieid 


Outputs 


'Turn  Fossa  Visible.NestedDM  ’  Page  1,  Mon,  Jun  6,  1994 


Inputs 


Synopsis: 

Surgeon  asks  nurses  to  pull  back 
skin,  muscle,  ribs. 


Open  Sesame 


per 
NestedDM 


Outputs 


Requirements: 

Speech  command  which  triggers 
opening  ol  abdominal  wall. 


Open  skint  <§>  just  a  test 


[  x-debug 


ate  User 


Library 


F-C  x-mstate  1 


NestedDM 


Cut  blood  off 


O oo 


NestedDM 


'2-Pull  Open.NestedDM  *  Page  1,  Mon,  Jun  6,  1994 


NestedDM 


EJ - ' 

Outputs 


'Open  Sesame. NestedDM  '  Page  1,  Mon,  Jun  6,  1994 


'Ghost  Scalpel. NestedOM  '  Page  1,  Mon,  Jun  6,  1994 


Each  ‘morph  control'  morphs 
a  different  side  of  the  skin. 


When  I  activate  both  morph  controls 
with  the  buttons,  BE  freezes  when  you 
move  the  scalpel.  .If  you  only  activate 
one.  it  works  fine. 


Packet  Data 


'tween.NestedDM  '  Page  1,  Mon,  Jun  6,  1994 


Sets  the  liver  to  the  flipped-up  state. 


'Surrogate  User.NestedDM  '  Page  1,  Mon,  Jun  6,  1904 


Inputs 

1  -  Objects  ths  Left  tool  can  grab  in  this  stats 

2  •  Whsthsr  ths  Left  tool  should  bs  abis  to  grab  right  now 

3  -  Objscts  ths  Right  tool  can  grab  in  this  stats 

4  •  Whsthsr  ths  Riaht  tool  should  bs  abis  to  arab  riaht  now 


Outputs 

1-  If  an  object  is  being  grabbed  by  ths  left  tool 

2-  What  object  is  being  grabbed  by  ths  left  tool 

3-  If  an  object  is  being  grabbed  by  the  right  tool 

4-  What  object  is  beino  crabbed  hv  the  riaht  tool 


Makes  the  tools  that  are  in  each  hand  follow  the 
index  finger. 


TooiGrabbing.Lib1  Page  1,  Mon,  Jun  6,  1994 


f  I  III 


'Switch  in  Dummy  EndrNaatadDM  '  Paga  1,  Mon,  Jun  6,  1994 


nrrr 


LULU 


Inputs 


Outputs 


'Rssst  for  15.NsstsdOM  *  Pag*  1,  Mon,  Jun  6,  1904 


Inputs 


Outputs 


'Reset  for  16.NestedDM  '  Page  1,  Mon,  Jun  6,  1904 


Inputs 


Outputs 


’Reset  for  17.NestedDM  '  Page  1,  Mon,  Jun  6,  1994 


Ini:  Which  state  are  you  in? 


Outl :  Goes  to  first  Fire  one  out. 
Out2:  Goes  to  second  Fire  one  out. 
Out3:  Goes  to  third  firs  one  out. 


'Switching  System.NestedDM  '  Page  1,  Mon,  Jun  6,  1994 


morph  the  two  pieces  of  skin  from  0  to  100  when  the 


scalpel  hits  them. 


This  should  be  changed  eventually  to  add  retractors 


'Open  8kin1.NestedDM  '  Page  1,  Mon,  Jun  6,  1994 


‘Cut  blood  off.  Nested  DM  *  Page  1,  Mon,  Jun  6,  1994 


